ML-Evaluation of Classifiers

Evaluation Criteria

Predictive accuracy:

$$Accuracy=\frac{Numberofcorrectclassification}{TotalNumberoftestcases}$$

Efficiency

• Time to construct the model
• Time to use the model

Robustness: handling noise and missing values Scalability: efficiency in disk-resident databases Interpretability: understandable and insight provided by the model Compactness of the model: size of the tree, or the number of rules.

**Precision and recall measures **

Confusion Matrix

	Classified Positive	Classified Negative
Actual Positive	TP	FN
Actual Negative	FP	TN

$$precision=\frac{TP}{TP+FP}$$$$recall=\frac{TP}{TP+FN}$$$$F_1=\frac{2\times precision}{precision+recall}$$

Roc Curve

True positive rate:

$$TPR=\frac{TP}{TP+FN}$$

False positive rate: / True Negative Rate

$$FPR=\frac{FP}{TN+FP}$$

How to compare 2 curves? Compute the area under the curve (AUC)

If AUC for $C_i$ is greater than that of $C_j$, it is said that $C_i$ is better than $C_j$. If a classifier is perfect, its AUC value is 1 If a classifier makes all random guesses, its AUC value is 0.5.

Evaluation Methods

Holdout set: The available data set D is divided into two disjoint subsets,

• the training set $D_{train}$ (for learning a model)
• the test set $D_{test}$ (for testing the model)

Important: training set should not be used in testing and the test set should not be used in learning.

• Unseen test set provides a unbiased estimate of accuracy.

The test set is also called the holdout set. (the examples in the original data set D are all labeled with classes.)

This method is mainly used when the data set D is large.

n-fold cross-validation:

The available data is partitioned into n equal-size disjoint subsets. Use each subset as the test set and combine the rest $n-1$ subsets as the training set to learn a classifier.

The procedure is run n times, which give n accuracies.

The final estimated accuracy of learning is the average of the n accuracies.

10-fold and 5-fold cross-validations are commonly used.

This method is used when the available data is not large.